DEMSort — Distributed External Memory Sort
نویسندگان
چکیده
We present the results of our DEMSort program in various categories of the SortBenchmark. DEMSort is a sophisticated and highly tuned implementation of a mergesort-based algorithm. It makes use of several libraries to support at the same time distributed-memory parallelism and shared-memory parallelism, in addition to very efficient disk I/O. This makes it excellent for sorting huge volumes of data. Our timings beat previous result by more than a factor of three, albeit using a much smaller machine.
منابع مشابه
Energy-Efficient Fast Sorting 2011
Authors of this report participated in the Sort Benchmark contests in 2009 and 2010. In 2009, our DEMSort program took the lead in the then-new Indy Gray category [RSSK09, RSS10], sorting 100 TB on a cluster with about 200 nodes. A tie was declared with Yahoo, whose Hadoop-based program achieved about the same result in the Daytona class, but with 17 times the hardware effort. Former results in...
متن کاملExternal Memory Sort On CGM1 Clusters
External memory sort has been widely accepted as an overall benchmark to evaluate the processing performance of computers. Lots of algorithms have been developed to sort large scalable data in different environments. The bottlenecks of external memory sort are the I/O operation and communication cost. In this paper, we adapted HPVM MinuteSort[4], and borrowed the THsort[5] idea to develop our e...
متن کاملParallel Merge Sort for Distributed Memory Architectures Parallel Merge Sort for Distributed Memory Architectures Ii Contents 1 Introduction 1 2 Cole's Parallel Merge Sort: the Crew Algorithm 1 3 Implementing the Crew Algorithm on a Distributed Memory Ar- Chitecture 7
Cole presented a parallel merge sort for the PRAM model that performs in O log n parallel steps using n processors He gave an algorithm for the CREW PRAM model for which the constant in the running time is small He also gave a more complex version of the algorithm for the EREWPRAM the constant factor in the running time is still moderate but not as small In this paper we give an approach to imp...
متن کاملI/O-Efficient Well-Separated Pair Decomposition and its Applications
We present an external memory algorithm to compute a well-separated pair decomposition (WSPD) of a given point set P in d in O sort N I/Os using O N B blocks of external memory, where N is the number of points in P, and sort N denotes the I/O complexity of sorting N items. (Throughout this paper we assume that the dimension d is fixed). We also show how to dynamically maintain the WSPD in O log...
متن کاملPatTrieSort - External String Sorting based on Patricia Tries
External merge sort belongs to the most efficient and widely used algorithms to sort big data: As much data as fits inside is sorted in main memory and afterwards swapped to external storage as so called initial run. After sorting all the data in this way block-wise, the initial runs are merged in a merging phase in order to retrieve the final sorted run containing the completely sorted origina...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2009